<html>
<head><meta charset="utf-8"><title>next few steps? · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html">next few steps?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="159628705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159628705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159628705">(Feb 28 2019 at 16:25)</a>:</h4>
<p>I know there is a plan, or at least a vague one, but I don't remember precisely what it is .. what are the next few steps we have in mind? <span class="user-mention" data-user-id="116466">@Zoxc</span> if nothing else maybe we can enumerate your pending PRs. =)</p>
<p>Do we have some kind of "criteria" that we want to have established before we stabilize? (Performance and otherwise)</p>
<p>I remember <span class="user-mention" data-user-id="123586">@nagisa</span> in particular raising concerns about using our own fork of rayon. We should probably be forming a plan for how to stop doing that, too. This was something <span class="user-mention" data-user-id="138448">@Josh Stone</span> and I had sort of hoped to have done at the Rust All Hands, but that didn't happen, although I think one of <span class="user-mention" data-user-id="138448">@Josh Stone</span>'s recent PRs would help somewhat here.</p>



<a name="159630399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630399">(Feb 28 2019 at 16:45)</a>:</h4>
<p>There's <a href="https://github.com/rust-lang/rust/issues/48685" target="_blank" title="https://github.com/rust-lang/rust/issues/48685">https://github.com/rust-lang/rust/issues/48685</a> which is probably rather out-of-date.</p>



<a name="159630470"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630470" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630470">(Feb 28 2019 at 16:46)</a>:</h4>
<p>I think setting a goal for what would be an acceptable overhead is a good idea.</p>



<a name="159630526"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630526" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630526">(Feb 28 2019 at 16:47)</a>:</h4>
<p>e.g. if the parallel compiler (restricted to a single thread) is within 5% of the current performance, then we can switch, or something like that</p>



<a name="159630555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630555">(Feb 28 2019 at 16:47)</a>:</h4>
<p>one of the bigger unknowns is also testing.</p>



<a name="159630639"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630639" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630639">(Feb 28 2019 at 16:48)</a>:</h4>
<p>I think we kind of agree that the regular tests should be run on a single thread and that they don't stress parallelism enough to be worth much anyway (because it's all small crates).</p>



<a name="159630700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159630700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159630700">(Feb 28 2019 at 16:49)</a>:</h4>
<p>we will get some degree of testing out of the normal bootstrapping process</p>



<a name="159631087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159631087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159631087">(Feb 28 2019 at 16:54)</a>:</h4>
<p>one question here is what we even want to test for. if it's just crashes then it wouldn't be to hard to have bors make sure that a certain set of large crates can be compiled without crashes or unexpected results.</p>



<a name="159631172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159631172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159631172">(Feb 28 2019 at 16:55)</a>:</h4>
<p>we could also do something like we did for incremental compilation: compile crates in both modes (1 and N threads) and then compare their output.</p>



<a name="159631259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159631259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159631259">(Feb 28 2019 at 16:56)</a>:</h4>
<p>(this was the tool in question: <a href="https://github.com/nikomatsakis/cargo-incremental" target="_blank" title="https://github.com/nikomatsakis/cargo-incremental">https://github.com/nikomatsakis/cargo-incremental</a>)</p>



<a name="159632175"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632175" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632175">(Feb 28 2019 at 17:07)</a>:</h4>
<p>I wonder if we could do some A/B testing, that is, compile every second nightly with <code>parallel-compiler=true</code> :)</p>



<a name="159632749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632749">(Feb 28 2019 at 17:14)</a>:</h4>
<p>Comparing crates with <code>parallel-compiler=false</code> and <code>parallel-compiler=true</code> seems like a good idea. Is our output deterministic enough for that?</p>



<a name="159632815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632815">(Feb 28 2019 at 17:15)</a>:</h4>
<p>maybe sorting the JSON output alphabetically would be enough?</p>



<a name="159632846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632846">(Feb 28 2019 at 17:15)</a>:</h4>
<p>binary output should be deterministic, I think</p>



<a name="159632870"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632870" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632870">(Feb 28 2019 at 17:15)</a>:</h4>
<blockquote>
<p>I think one of @_<strong>Josh Stone</strong>'s recent PRs would help somewhat here.</p>
</blockquote>
<p>Said PR: <a href="https://github.com/rayon-rs/rayon/pull/636" target="_blank" title="https://github.com/rayon-rs/rayon/pull/636">https://github.com/rayon-rs/rayon/pull/636</a> -- could replace rustc-rayon's main-handler and scoped threadpool</p>



<a name="159632960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159632960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159632960">(Feb 28 2019 at 17:16)</a>:</h4>
<p>The other big thing in rustc-rayon was the thread-local stuff, especially the one that follows stolen jobs</p>



<a name="159633049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159633049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159633049">(Feb 28 2019 at 17:17)</a>:</h4>
<p>I'm not sure we should get rid of <code>cfg!(parallel_compiler)</code>. That is a useful tool to measure the overhead of parallelization, and isn't overly invasive.</p>



<a name="159633512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159633512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159633512">(Feb 28 2019 at 17:23)</a>:</h4>
<blockquote>
<p>I'm not sure we should get rid of cfg!(parallel_compiler). </p>
</blockquote>
<p>We don't need to rush it. In general I'm not a big fan of keeping it around forever because it makes things more complicated.</p>



<a name="159633611"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159633611" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159633611">(Feb 28 2019 at 17:24)</a>:</h4>
<p>But yes, it's not too bad and I think defaulting to parallel-compiler is the actual goal.</p>



<a name="159633694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159633694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159633694">(Feb 28 2019 at 17:25)</a>:</h4>
<p>I'll change the wording in the PR to make this clear.</p>



<a name="159633787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159633787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159633787">(Feb 28 2019 at 17:26)</a>:</h4>
<p>(done)</p>



<a name="159634346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159634346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159634346">(Feb 28 2019 at 17:36)</a>:</h4>
<p>My mental plan was to<br>
- Land <a href="https://github.com/rust-lang/rust/pull/58679" target="_blank" title="https://github.com/rust-lang/rust/pull/58679">https://github.com/rust-lang/rust/pull/58679</a> and <a href="https://github.com/rust-lang/rust/pull/58019" target="_blank" title="https://github.com/rust-lang/rust/pull/58019">https://github.com/rust-lang/rust/pull/58019</a><br>
- Run crater, evaluate perf<br>
- Enable by default on nightly</p>



<a name="159635146"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159635146" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159635146">(Feb 28 2019 at 17:47)</a>:</h4>
<p>I also have a local branch which does parallel parsing, there's a race condition in it though</p>



<a name="159742092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/next%20few%20steps%3F/near/159742092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/next.20few.20steps.3F.html#159742092">(Mar 01 2019 at 19:36)</a>:</h4>
<p>(thanks all, this was enlightening to read)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>